1. Field of the Invention
The present invention relates, generally, to the communication of data in a storage system, and in particular embodiments, to preventing head of line blocking in a multi-rate switched Fibre Channel (FC) fabric loop attached system.
2. Description of Related Art
As illustrated in the exemplary interconnection diagram of a storage system 100 shown in FIG. 1, non-blocking frame-based buffered crossbar switches 102 and 152 enable a “fabric” interconnection of a large number of devices such as Host Bus Adapters (HBAs) 104 and 106, and groups of disk drives referred to as “don't care” Bunches Of Disks (xBODs) 108, 110, 112, 114 and 116. The devices are connected to ports in the switches 102 and 152. Each port contains a predetermined number of receive (Rx) buffers (e.g. five) for storing data received into the port, and a predetermined number of transmit (Tx) buffers (e.g. five) for storing data to be transmitted out of the port. In FIG. 1, port P0 of buffered switch 102 is connected to HBA 104, port P1 is connected to HBA 106, port P2 is connected to xBOD 108, port P3 is connected to xBOD 110, and N-Port P4 is connected to cascaded switch 152. Port P1 of switch 152 is connected to xBOD 116, port P2 is connected to xBOD 112, port P3 is connected to xBOD 114, and port P4 is connected to xBOD 154.
Note that unlike Fibre Channel (FC) arbitrated loop (AL) storage switches, which utilize an 8-bit Arbitrated Loop Protocol Address (AL_PA), have a 126 device limit, connect and switch loop devices that must share the bandwidth, and cannot be connected to any other storage switch, frame-based buffered switches 102 and 152 utilize a 24-bit address (which includes 8-bit domain and area fields in addition to an 8-bit AL_PA), have a much higher device limit, and connect and switch devices that do not have to share the bandwidth. Frame-based buffered switches enclosures support loop devices such as disk drives in an xBOD connected via a port, and can also be connected to other frame-based buffered switches via an inter-switch link.
Both HBAs and xBODs operate at up to a particular maximum line or signaling rate, such as 2 Gbits/sec or 4 Gbits/sec. An alternative form of measurement would be throughput. In general, a destination device (e.g. an HBA) with a lower line rate or throughput (referred to herein as a low speed destination device) that is attempting to access the same source device (e.g. an xBOD) with a higher line rate or throughput (referred to herein as a high speed source device) an as another destination device (e.g. another HBA) with a higher line rate or throughput (referred to herein as a high speed destination device) may result in the limiting or throttling of the overall throughput of the system to that of the low speed destination device. This is commonly referred to as the “multi-data rate head of line blocking” problem.
In the example of FIG. 1, suppose that HBA 104 operates at a 4 Gbits/sec, HBA 106 operates at 2 Gbits/sec, xBODs 108, 116 and 154 operate at 4 Gbits/sec, and xBODs 110, 112 and 114 operate at 2 Gbits/sec. The problem occurs when an HBA with a low speed line rate (e.g. HBA 106) and an HBA with a high speed line rate (e.g. HBA 104) both attempt to read data from the same source device having a high speed line rate. For example, in FIG. 1 the problem would occur if both HBAs 104 and 106 were both attempting to read data from xBOD 154, or alternatively xBODs 112 and 114 aggregated together, through port P4 of switch 102, which results in a 4 Gbits/sec signaling rate at port P4 (and may be referred to herein as a single source device). When data destined for HBA 106 (the low speed HBA) arrives at P4, it is first placed in receive buffers 118. When a connection through switch core 120 to P1 is made, the data is transferred to transmit buffers 122 of P1 prior to final transmission to HBA 106. However, because HBA 106 is slower than the effective signaling rate seen at port P4, the transmit buffers 122 will fill up more quickly than they can be drained. The system becomes blocked by the HBA 106 when sufficient data arrives from P4 to fill all of the transmit buffers 122 plus at least one receive buffer 118 in P4 of switch 102. At this point, although data received into P4 and destined for HBA 104 can be stored in any empty receive buffers 118, this data will be blocked by the data already stored in the receive buffers 118 and destined for HBA 106, and will remain blocked until the data for HBA 106 is drained from the receive buffers 118 at its low speed rate. Only at that time can the data destined for HBA 104 be forwarded to the transmit buffers 124 of P0. The net effect of the blocking is to reduce the throughput of both HBAs 104 and 106 closer to the speed of the slowest HBA 106.
Note that the problem does not occur when an HBA with a low speed line rate (e.g. HBA 106) and an HBA with a high speed line rate (e.g. HBA 104) both attempt to read data from the same source device having a low speed line rate. For example, in FIG. 1 the problem will not occur if both HBAs 104 and 106 were both attempting to read data from xBOD 110 (with a 2 Gbits/sec signaling rate at port P3 of switch 102). When data destined for HBA 106 (the low speed HBA) arrives at P3, it is first placed in receive buffers 134. When a connection through switch core 120 to P1 is made, the data is transferred to transmit buffers 122 of P1 prior to final transmission to HBA 106. However, because HBA 106 is the same speed as xBOD 110, the transmit buffers 122 will be drained as quickly as they can be filled up. Because the receive buffers 134 do not get backed up with data destined for P1, the system does not become blocked, and data received into P3 and destined for HBA 104 can be stored into any empty receive buffers 134 for subsequent transfer to the transmit buffers 124 of P0.
The blocking situation as described above occurs when read commands from HBAs of different speeds are issued to xBODs attached to the same source port (and must therefore share the same source port receive buffers). However, a similar blocking situation occurs when write commands from an HBA are issued to xBODs of different speeds attached to different ports on the same switch. In the example of FIG. 1, if HBA 104 (a high speed HBA) issued a write request to both xBOD 108 (a high speed xBOD) and xBOD 110 (a low speed xBOD), because xBOD 110 is a slow speed device, data being written to xBOD 110 may get backed up in the source port receive buffer 150, preventing HBA 104 from writing data to xBOD 108.
Therefore, there is a need to reduce the number of occurrences of head of line blocking so that a low speed destination device does not block the flow of data to a high speed destination device when both devices are attempting to access a high speed source device.